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I.  INTRODUCTION 


Flight  simulators  are  of  critical  importance  in  the 
training  of  combat  pilots.  In  this  regard,  data 
displayed  to  the  instructor/operator  of  the  flight  simu¬ 
lator  should  cha ra c ter i ze . the  performance  succinctly  to 
facilitate  the  instructor's  tasks.  These  include 
measuring  the  proficiency  demonstrated  and  producing  a 
medium  for  superior  instruction.  The  purpose  of  this 
effort  was  to  develop  specific  techniques  that  are  suit¬ 
able  for  the  analysis  and  characterization  of  tactical 
performance  in  a  flight  simulator. 

The  techniques  are  based  on  the  use  of  the  adaptive 
maneuver  logic  (AML)  (Reference  1),  a  computer  program 
developed  to  act  as  a  real-time,  interactive  opponent  on 
flight  simulators.  The  AML  program  decides  its  moves  on 
a  second- by -second  basis  by  examining  its  relation  to  its 
opponent.  At  each  decision  point,  the  program  projects 
its  own  position  three  seconds  into  the  future,  based  on 
executing  each  of  several  hypothetical,  al ternati ve  moves . 
It  also  projects  its  opponent's  position  and  then  computes 
the  answers  to  a  number  of  yes  or  no  questions  (e.g., 

"Will  I  be  above  my  opponent?",  "Will  I  be  behind  my 
opponent?").  Each  question  is  weighted  by  a  number  that 
reflects  its  approximate  value  in  combat  flying.  For 
each  move,  the  program  adds  up  the  weights  of  the  ques¬ 
tions  answered  "yes."  It  chooses  the  move  with  the 
highest  sum  as  the  next  move  to  make. 

The  weights  applied  to  each  question  determine  the 
quality  of  flight  by  the  AML.  One  important  aspect  of 
skilled  performance  is  having  the  correct  internal  repre¬ 
sentation  of  the  relative  importance  or  weights  for  ques¬ 
tions  such  as  these.  With  a  capability  of  inferring 
these  weights  from  recorded  or  incoming  flight  data, 
novel  information  could  be  displayed  to  the  instructor 
about  wh£t  the  pilot  is  emphasizing  and  trying  to  achieve 
when  performing  combat  tactics.  The  approach  here  was  to 
develop  such  a  capability  by  using  a  modified  AML  program 
Specifically,  this  AML  was  expected  to  compute  the  set  of 
weights  that  allowed  it  to  most  closely  model  the  pilot's 
performance.  It  was  theorized  that  these  weights,  or  at 
least  the  relative  values,  would  approximate  those  of  the 
pilot. 

In  Phase  I  (Reference  2)  of  this  effort,  basic 
computing  techniques  were  developed  using  data  generated 
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by  one  AML  flying  against  another.  Given  a  record  of 
performance  on  a  second-by-second  basis  of  the  AML,  a  set 
of  weights  was  computed  by  a  modified  AML  program  which, 
if  used  by  the  first  AML,  would  cause  it  to  duplicate  the 
observed  flight.  Phase  II  concerned  the  use  of  these 
techniques  for  the  purpose  of  modelling  the  flight 
performance  behavior  exhibited  by  human  pilots. 

Part  of  the  Phase  I  work  included  a  reexamination  of 
the  questions  used  by  AML  to  assure  that  they  would 
provide  meaningful  data  to  an  instructor.  It  was  decided 
that  the  questions  needed  to  be  revised  to  reflect  infor¬ 
mation  on  how  well  the  pilot  performs  the  maneuver  in 
question.  A  further  decision  was  to  restrict  attention  to 
a  single  maneuver  --  the  high-speed  yo-yo.  Work  in  Phase 
II  concentrated  on  implementing  the  new  set  of  questions, 
obtaining  good  reference  flights  from  instructor  pilots, 
getting  the  AML  to  fly  a  good  high-speed  yo-yo,  collecting 
student  data,  and  using  AML  to  compute  weights  (or 
importances)  for  the  questions  for  each  student. 

Although  Initially  the  method  showed  promise,  success  failed  to 
materialize  because  of  a  lack  of  repeatability  in  human  student  pilot 
actions. 
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II.  DISCUSSION 


The  question  was  raised  whether  the  AML  with  its 
relative  geometry  criteria  could  simulate  the  pilot 
flying  different  standard  air  combat  maneuvers .  It  was, 
therefore,  proposed  that  for  Phase  II  the  flights  be 
restricted  to  a  single  maneuver  but  not  a  simple  one. 

The  one  chosen  was  a  high-speed  yo-yo  against  a  nonin¬ 
teractive  target  flying  a  defensive  turn.  For  a  given 
initial  condition  and  noninteractive  target,  a  good 
reference  high-speed  yo-yo  should  be  developed,  and  the 
criteria  in  the  question  set  should  involve  this  refer¬ 
ence  yo-yo.  This  should  provide  meaningful  data  to  the 
instructor  and  force  the  modified  AML  to  fly  a  high-speed 
yo-yo  approximating  the  reference  yo-yo.  Finally,  the 
adequacy  of  the  set  of  trial  maneuvers  was  reexamined  and 
the  necessary  data  collected  at  Luke  AFB.  These 
activities  are  described  next. 


Question  Set 

In  determining  an  adequate  question  set,  two  points 
had  to  be  considered.  First,  the  information  contained  in 
the  questions  should  be  adequate  for  the  short-term 
maneuver  logic  of  the  AML  which  requires  relative  geometry 
and  dynamics  information  for  the  AML  plane.  On  the  other 
hand,  since  the  AML  is  to  be  used  as  a  basis  for  measuring 
the  flight  performance  behavior  of  human  pilots,  the 
questions  must  be  meaningful  to  a  pilot.  They  should 
contain  variables  directly  displayed  to  the  pilot  or 
relatively  easily  perceived  by  the  pilot.  On  the  basis 
of  these  criteria,  the  following  set  of  eight  questions 
was  devel oped  : 

1.  Is  my  relative  altitude  correct? 

2.  Is  my  relative  altitude  rate  correct? 

3.  Is  my  relative  velocity  correct? 

4.  Is  my  load  factor  correct? 

5.  Is  my  range  correct? 

6.  Is  my  range  rate  correct? 

7.  Is  my  1 i ne-of - s i ght  angle  (LOS)  correct? 
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8.  Is  my  throttle  setting  correct? 


The  criteria  for  correct  values  were  to  be  taken 
from  the  reference  yo-yo  maneuvers.  In  getting  the  AML 
to  fly  a  high-speed  yo-yo,  the  following  procedure  was 
used.  For  each  question,  the  absolute  value  of  the 
difference  between  actual  and  reference  values  was  calcu¬ 
lated  and  normalized  by  the  maximum  error  in  that 
question.  This  was  then  multiplied  by  a  weight  factor 
which  remained  constant  during  the  entire  maneuver.  The 
total  value  V  for  a  given  trial  maneuver  was  then: 


REFERENCE  VALUE^  -  ACTUAL  VALUE i 
MAX  ERR0Ri 


where  W-j  was  the  weight  factor  associated  with  the  ith 
question.  At  each  decision  point,  a  value  V  was  then 
determined  for  each  trial  maneuver,  and  the  trial  maneuver 
with  the  highest  value  V  was  chosen  as  the  next  maneuver 
for  the  AML. 


Trial  Maneuvers 


We  also  had  to  consider  the  question,  "What  is  a 
good  set  of  trial  maneuvers  for  a  high-speed  yo-yo?"  The 
original  set  of  trial  maneuvers  in  the  AML  consisted  of 
a  continuation  of  the  present  flight,  a  straight  flight 
or  maximum-g  turns  in  the  two  planes  (maneuver  planes) 
that  form  plus  or  minus  the  tilt  angle  (an  input  para¬ 
meter)  with  the  base  plane  (the  plane  defined  by  the  AML 
airplane's  velocity  vector  and  the  opponent's  extrapo¬ 
lated  position).  It  was  decided  to  add  four  more  trial 
maneuvers  to  this  set.  These  were  four  maximum-g  turns 
in  the  planes  which  form  plus  or  minus  twice  and  three 
times  the  tilt  angle  with  the  base  plane.  Also,  since 
the  tilt  angle  is  an  input  parameter  to  the  AML,  the 
value  of  it  could  be  changed  between  runs,  thus, 
obtaining  different  sets  of  trial  maneuvers,  if  desired. 

For  the  original  AML,  the  base  plane  defined  by  the 
AML  airplane's  velocity  vector  and  the  opponent’s  extra¬ 
polated  position  gave  trial  maneuvers  which  allowed  the 
AML  considerable  success  in  getting  in  a  cone  behind  the 
opponent.  While  this  position  is  the  goal  of  the 
attacker  in  a  high-speed  yo-yo  after  the  apex,  it  is  not 
so  for  that  part  of  the  flight  prior  to  the  apex. 

Further,  since  the  purpose  here  is  basically  to  fly  a 


8 


yo-yo  close  to  the  reference  yo-yo,  it  was  decided  that  a 
better  base  plane  would  be  the  plane  defined  by  the  AML 
airplane's  velocity  vector  and  the  projected  position  of 
the  reference  yo-yo. 


Reference  Yo-Yo 


Finally,  a  set  of  reference  data  for  good  high-speed 
yo-yo  maneuvers  had  to  be  obtained.  Collection  of  these 
data  was  accomplished  at  Luke  AFB  on  the  Simulator  for 
Air-to-Air  Combat  (SAAC)  simulator.  Initially,  12 
different  initial  conditions  (I/Cs)  were  considered. 

After  some  trial  runs  and  discussions  with  the  instructor 
pilots  at  Luke  AFB  it  was  decided  to  have  only  two 
different  I/Cs  and  have  each  of  the  two  instructor  pilots 
fly  three  runs  for  each  I/C.  These  I/Cs  were  chosen  so 
that  immediate  execution  of  a  high-speed  yo-yo  would  be 
indicated.  The  I/Cs  were  angle-off's  of  50  degrees  and 
60  degrees  to  the  target  aircraft,  550  KIAS  velocity, 
range  of  8,000  feet,  and  altitude  of  15,000  feet.  The 
target  aircraft  had  velocity  of  375  KIAS,  altitude  of 
15,000  feet,  and  was  to  fly  a  3g  defensive  turn. 

The  flightpath  data,  recorded  in  32-bit  words  in 
Sigma  5  format,  were  then  converted  to  48-bit  words 
compatible  with  the  AML  program  on  the  CDC  3600  computer 
and  stored  on  magnetic  tape.  The  data  were  checked  for 
the  number  of  files  (i.e.,  the  number  of  flights)  and 
within  each  file  or  flight  the  number  of  blocks  of  data. 

A  block  of  data  was  recorded  every  half-second  during  the 
flight.  This  file  information  was  maintained  so  that  any 
given  segment  of  flight  data  could  be  read  from  the 
magnetic  tape. 

Using  plot  routines,  the  following  functions  of  time 
in  the  flight  were  plotted  for  each  yo-yo:  (a)  line-of- 
sight  (LOS)  angle,  (b)  range  and  range  rate,  (c)  mach 
number,  (d)  altitude,  and  (e)  ground  trace.  Since  one  of 
the  team  members  is  a  former  fighter  pilot,  it  was 
decided  to  have  him  rate  the  different  yo-yo  maneuvers  on 
the  basis  of  these  plots.  For  rating  purposes,  the 
maneuver  was  considered  completed  when  the  3,000-foot 
range  point  was  reached. 

Criteria  used  in  rating  the  runs  included  minimum 
undershoot/overshoot,  maximum  altitude  difference  at  the 
apex,  range  rate  decreasing  up  to  the  point  of  maximum 
altitude  difference  and  then  increasing  until  the  3,000- 
foot  "in-range"  point,  minimum  range  rate  at  the  apex  and 
LOS  angle  approaching  0  at  3,000-foot  range.  Each  flight 
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was  scored  on  each  plot  as  well  as  on  the  elapsed  time  to 
completion.  The  scores  for  each  flight  were  summed  and 
the  flights  ranked  by  total  score. 


Run  #14  was  rated  the  best  for  the  first  instructor 
pilot  and  run  #22  for  the  second,  with  run  #14  rated  best 
overall.  These  two  runs  were  used  as  reference  runs. 
Figures  1  and  2  are  the  altitude  and  ground  trace  plots 
for  run  #14,  and  Figures  3  and  4  are  for  run  #22. 


Normal  Acceleration 


I 


In  the  Phase  I  part  of  this  study,  it  was  discovered 
that  the  recorded  acceleration  along  the  body  z-axis,  az, 

was  not  always  in  agreement  with  that  computed  by  the  AML 
using  recorded  flightpath  data.  Since  load  factor  (i.e, 
acceleration  along  body  z-axis  plus  component  of  gravity 
vector  along  it)  is  a  required  input  to  the  AML  program, 
it  is  essential  that  the  load  factors  agree.  Two  listings 
of  aircraft  simulation  programs  were  looked  over.  In 
both,  the  accelerations  were  computed  in  the  body  axis 
coordinates  rather  than  in  inertial  coordinates  as  in  the 
AML.  The  results  should  be  the  same,  but  it  was  decided 
to  compute  them  in  the  body  axis  as  a  check. 

Now  the  acceleration,  az  ,  along  the  z-axis  in  body 
axes  coordinates  is  given  by 

■  3T  *  pv  -  <!" 


where  U,  V,  and  W  are  the  velocity  components  along  the 
x,  y,  and  z  body  axes,  and  P  and  Q  are  the  roll  and  pitch 
rates.  az  was  computed  by  this  formula  at  various  points 

in  the  flightpaths  for  both  the  attacker  and  target 
aircraft.  The  computed  results  for  the  target  agreed  well 
with  the  recorded  az  over  all  parts  of  the  flightpath, 

but  the  computed  results  did  not  agree  with  the  recorded 
az  throughout  the  entire  flightpath  of  the  attacker. 

During  the  portion  of  the  flight  were  the  attacker  is 
climbing  and  turning  rapidly,  the  recorded  az  is  much 

larger  than  the  computed  load  factor;  for  example,  at  10 
seconds  the  recorded  az  is  5g  while  the  computed  load 

factor  is  4 . 1 9g .  In  order  to  have  compatible  load 
factors,  it  was  decided  to  compute  them  from  the  recorded 
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flightpath  data.  The  radius  of  curvature  was  used  to 
compute  the  load  factor  by  the  formula. 


load  factor  =  V2/R  =  g  cos  0  cos  t> 

where  V  is  velocity  of  aircraft,  R  is  radius  of  curvature, 
0  is  pitch  angle,  and  <}>  is  roll  angle. 

The  radius  of  curvature  is  calculated  as  the  radius 
of  the  circle  through  the  present  position  of  the  plane 
and  previous  and  next  positions.  If  the  three  positions 
are  ( x  ,  y  ,  z  ) ,  ( x  ,  y  ,  z  ) ,  and  (x  ,  y  ,  z  ),  then 

111  2  2  2  1  1  t 

the  center  of  the  circle  (x,  y,  z)  is  the  solution  of  the 
following  three  equations: 

2(x  -x  )x+2(y  -y  )y+2(z  -z  )z  =  x2 +y2 +z2  -  (  x2 +y2 +z2  ) 
21  21  21  2  2  2  111 

2 ( x  -x  )x  +  2(y  -y  )y  +  2(z  -z  )z  =  x? +y? +z? - ( x1 +ys +z !  ) 

32  32  32  33322? 

ax  +  by  +  cz  =1 

where  (a,  b,  c)  is  the  solution  set  of 


v?  =  x?  +  y?  + 

2  2  2 

A  subroutine  was  then  written  to  read  the  positional 
and  velocity  data  from  the  magnetic  tape  and  do  the  above 
computa t i on  for  the  load  factor  at  each  point  in  the 
flightpath.  This  subroutine  was  incorporated  into  the 
AML  program. 


AML  Evaluation  Run 


The  original  AML  program  was  run  against  the  target. 
Two  runs  were  made,  but  neither  was  a  high-speed  yo-yo 
maneuver.  When  the  modified  AML  was  run  against  the 
target,  it  did  fly  a  high-speed  yo-yo  similar  to  the 
reference  yo-yo.  Several  runs  were  made  using  different 
weights  and  tilt  angles,  but  except  for  runs  with  some 
weights  set  to  zero,  there  were  no  noticeable  differences 
in  the  maneuvers. 

During  these  runs,  it  was  discovered  that  the 
program  did  not  update  the  velocity  during  the  trial 
maneuvers.  This  problem  was  corrected.  (A  more  detailed 
account  of  this  section  is  given  in  Appendix  A.) 


Ree xami nati on  of  Question  Set 

Since  varying  the  weights,  other  than  setting  them 
to  zero,  did  not  really  yield  different  hiqh-speed  yo-yo 
maneuvers,  the  questions  were  reexamined.  For  the  trial 
maneuvers,  the  throttle  setting  depends  on  the  initial 
conditions  of  the  trial  maneuver  and  so  is  the  same  for 
all  trial  maneuvers  at  a  given  decision  point.  The  load 
factor  is  a  parameter  dependent  on  other  factors  at  the 
start  of  the  trial  maneuver  and  is  constant  throughout 
the  maneuver.  Therefore,  it  was  decided  to  delete  the 
questions  on  throttle  setting  and  load  factor.  Further, 
since  altitude  is  a  function  of  altitude  rate  and 
initial  conditions,  and  range  is  a  function  of  range 
rate  and  initial  conditions,  they  are  somewhat  redundant . 
Also,  the  questions  considered  only  the  magnitude  (abso¬ 
lute  value)  of  the  error  ignoring  whether  the  error  was 
high  or  low.  In  addition,  some  of  the  original  questions 
of  the  AML  program  appear  to  be  apropos  to  the  high-speed 
yo-yo.  Based  on  these  observa t ions  ,  the  following  set 
of  questions  was  developed  and  incoporated  in  the  AML 
program. 


1 . 

Is 

opponent  in  front 

of 

me? 

2. 

Can 

I  see  opponent? 

3. 

Can 

opponent  not  see 

me? 

4  . 

Is 

my  al ti tude  rate 

no  t 

too 

low? 

5. 

Is 

my  altitude  rate 

not 

too 

high? 

6. 

I  s 

my  range  rate  not 

too 

1  ow? 
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7. 

Is 

my 

range  rate  not  too  high? 

8. 

Is 

my 

velocity  not  too 

low? 

9. 

Is 

my 

velocity  not  too 

high? 

10. 

Is 

my 

LOS  angle  within 

limits? 

11. 

Is 

my 

LOS  angle  rate  favorable? 

For  altitude  rate,  range  rate,  velocity,  and  LOS 
angle,  envelopes  were  developed  around  the  corresponding 
reference  values  as  functions  of  time  and  were  read  in 
as  data.  An  affirmative  answer  is  given  for  "not  too  low" 
if  the  value  for  the  parameter  is  above  the  lower  limit 
of  its  envelope  and  for  "not  too  high"  if  the  value  is 
below  the  upper  limit  of  its  envelope.  LOS  angle  is 
within  limits  if  it  is  between  the  upper  and  lower  limits 
of  its  envelope.  LOS  angle  rate  is  favorable  if,  when 
LOS  angle  is  high,  the  rate  is  negative,  if,  when  LOS 
angle  is  low,  the  rate  is  positive,  or  if,  when  LOS  angle 

is  within  limits,  the  rate  will  not  take  it  out  of  the 

envelope  within  the  next  second. 

A  run  was  made  with  reference  flight  #14  and  weights 
all  equal  to  5.  The  result  was  a  high-speed  yo-yo  but, 
again,  with  a  lower  apex  and  also  with  the  apex  occurring 
2  seconds  earlier  than  the  referenced  yo-yo.  Plots  are 
given  in  Figures  5  and  6. 

As  reported  earlier,  since  the  purpose  was  to  model 

the  referenced  high-speed  yo-yo,  it  was  decided  to  use  as 

base  plane  in  the  trial  maneuvers  the  plane  determined 
by  the  velocity  vector  and  the  projected  position  of  the 
referenced  high-speed  yo-yo.  This  was  implemented  in  the 
program.  Runs  were  then  made  with  run  #14  as  reference, 
with  the  new  question  set,  and  with  the  new  trial 
maneuvers.  Weights  were  all  5 ' s  and  tilt  angle  was  10 
degrees.  Prediction  times  (times  into  the  future  for 
which  the  student  flights  were  extrapolated)  were  2 
seconds  and  3  seconds.  Both  runs  had  slightly  higher 
apexes  with  the  3-second  run  being  the  better  of  the  two. 
Due  to  a  nonrecoverabl e  tape  error,  the  plot  data  for  the 
3-second  run  were  lost.  Plots  for  the  2-second  run  are 
given  in  Figures  7  and  8. 

While  the  new  question  set  was  not  a  decided 
improvement  over  the  first  set  of  questions  in  matching 
the  reference  yo-yo  maneuvers,  it  was  more  amenable  to 
application  of  the  techniques  developed  in  Phase  I  for 
deriving  an  independent  set  of  inequalities  from  the  AML 
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Figure  7.  Ground  trace.  AML  versus  LUKE  #14. 
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program  and  solving  them  by  linear  programming.  It  was, 
therefore,  used  in  running  the  AML  with  the  student  pilot 
flight  data  to  determine  the  apparent  weights  of  the 
student  pilot  with  respect  to  the  questions. 


Student  Pilot  Data 


Data  were  obtained  from  20  high-speed  yo-yo  maneuvers 
flown  by  five  student  pilots  (4  flights  each)  on  the  SAAC 
simulator  against  defensive  turns  flown  by  an  instructor 
pilot.  Initial  conditions  for  all  runs  were:  angle  off 
of  50  degrees,  velocity  of  550  KIAS,  range  of  8,000  feet, 
and  altitude  of  15,000  feet.  The  target  aircraft  had  375 
KIAS  velocity  and  altitude  of  15,000  feet  and  was  to  fly  a 
3g  defensive  turn.  The  data  were  converted  into  the 
format  necessary  for  use  in  the  AML  program  on  the  CDC 
3600. 


At  each  decision  point  (every  second),  the  AML 
program  was  reinitialized  to  the  current  position  of  the 
student  pilot's  flight  from  which  position  it  set  up  the 
trial  maneuvers.  For  each  of  the  reference  runs,  two  sets 
of  inequalities  were  derived;  one  in  which  the  student 
pilot's  position  at  the  prediction  time  was  compared  with 
all  the  trial  maneuvers  and  a  second  in  which  it  was 
compared  only  with  the  best  trial  maneuver  (best  in  the 
sense  that  it  was  the  trial  maneuver  which  had  the  most 
questions  answered  affirmatively).  Hence,  for  each  of  the 
20  student  flights,  4  different  sets  of  weights  were 
computed . 

In  Phase  I,  weights  in  the  LP  program  were  restricted 
to  be  between  1  and  5.  Since  the  AML  is  deterministic  and 
consistent  in  its  decisions,  solution  with  these  restric¬ 
tions  always  existed.  Hence,  minimum  and  maximum  solution 
could  be  found  which  gave  bounds  on  the  possible  solutions; 
however,  when  the  LP  program  with  these  restrictions  was 
run  on  the  student  pilot  data,  no  solution  existed.  This 
means  that  for  the  student  pilots  it  is  not  possible  to 
find  a  set  of  positive  weights  that  will  satisfy  the  set 
of  inequalities.  The  requirement  that  the  weights  be 
greater  than  or  equal  to  one  had  to  be  dropped  and  zero 
weights  admitted.  The  minimum  solution  then  consists  of 
all  zeros  and  does  not  add  any  information.  Hence,  only 
the  maximum  solution  was  obtained  by  the  LP  program. 


Summary 

Information  in  the  question  set  should  be  adequate 
for  the  AML  to  fly  a  high-speed  yo-yo  maneuver  and  should 
also  be  meaningful  to  the  pilot.  It  should  be  directly 
displayed  or  relatively  easily  perceived  by  the  pilot. 
Further,  the  criteria  for  correct  values  should  involve 
the  reference  yo-yo's.  On  the  basis  of  these  requirements 
the  above  list  of  questions  was  developed. 

Four  new  maneuvers  were  added  to  the  set  of  trial 
maneuvers,  and  the  base  plane  was  changed  to  be  the  plane 
defined  by  the  AML's  velocity  vector  and  the  projected 
position  of  the  reference  yo-yo. 

Using  initial  conditions  that  would  require  the  imme¬ 
diate  execution  of  a  high-speed  yo-yo  maneuver,  two 
instructor  pilots  on  the  SAAC  at  Luke  AFB  each  flew  six 
runs  against  a  noninteractive  target  flying  a  3g  defensive 
turn.  The  best  high-speed  yo-yo  maneuver  of  each  pilot 
was  chosen  as  a  reference  yo-yo. 

In  Phase  I,  it  was  discovered  that  the  recorded 
normal  acceleration  along  the  z-axis  did  not  always  agree 
with  that  computed  by  the  AML  using  associated  recorded 
flightpath  data.  This  disagreement  also  occurred  in  the 
high-speed  yo-yo  data.  To  solve  this  problem  it  was 
decided  to  use  computed  normal  acceleration  rather  than 
the  recorded  data.  A  subroutine  was  written  to  compute 
the  normal  acceleration  from  positional  and  velocity  data 
of  the  yo-yo  maneuver  and  was  incorporated  into  the 
program . 

Both  the  original  and  the  modified  AML  program  were 
run  against  the  target  airplane.  The  original  AML  did 
not  fly  a  high-speed  yo-yo,  but  the  modified  program  did. 
Several  runs  were  made  by  the  modified  AML,  but  it  was 
not  sensitive  to  weight  changes  except  when  some  weights 
were  set  to  zero. 

An  analysis  showed  that  two  parameters  in  the  ques¬ 
tion  set  were  not  updated  in  the  trial  maneuvers  and  so 
did  no  differentiating,  that  other  parameters  had  redun¬ 
dancies  in  them  and  that  the  questions  considered  only 
the  magnitude  of  the  error  ignoring  whether  the  error 
was  high  or  low.  A  new  set  of  questions  were  developed 
and  tested.  While  it  was  not  a  decided  improvement  over 
the  old  set,  it  was  more  amenable  to  the  techniques 
developed  in  Phase  I.  It  was  decided  to  use  the  new  set. 
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Data  were  obtained  from  20  high-speed  yo-yo  maneu¬ 
vers  flown  by  student  pilots  on  the  SAAC  simulator  with 
the  same  initial  conditions  as  for  the  reference  yo-yo 
maneuvers.  Five  pilots  each  flew  4  maneuvers.  By  having 
the  AML  reinitialized  at  each  decision  point  to  the 
current  position  of  the  student's  flight  from  which  point 
it  set  up  the  trial  maneuvers,  the  AML  was  able  to  compare 
the  student  pilot's  projected  position  with  that  of  each 
trial  maneuver.  This  gave  rise  to  sets  of  inequalities 
from  which  the  student  pilot's  weights  coul d  be  determi ned . 
Unlike  the  AML,  the  student  pilots  were  not  deterministic 
so  that  nonzero  solutions  to  the  inequalities  did  not 
exist.  Zero  weights  had  to  be  allowed  in  order  to  get 
solution  sets. 
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III.  EXPERIMENTAL  RESULTS 


Each  student  flight  was  run  twice  against  each  of  the 
reference  flights,  once  comparing  the  student  pilot's 
position  at  prediction  time  with  all  trial  maneuvers  and 
second,  comparing  it  only  with  the  best  trial  maneuver. 
Hence,  four  sets  of  inequalities  and,  therefore,  four  sets 
of  weights  were  obtained  for  each  of  the  20  student 
flights.  The  weights  for  each  set  obtained  using  the 
maximum  solution  of  the  LP  program  are  given  in  Table  1. 

To  simplify  the  table,  only  the  questions  with  nonzero 
weights  are  listed.  Unless  indicated,  weights  are  assumed 
to  be  5.  Student  flight  #12  with  respect  to  reference 
flight  #14  and  using  the  best  trial  maneuver  for  compar¬ 
ison  had  nonzero  weights  of  5  and  2.5.  Those  questions 
having  weights  of  5  are  listed  in  the  first  row,  and  those 
with  weights  of  2.5  are  listed  in  the  second  row.  The 
same  is  done  for  flight  #15  with  reference  flight  #22  and 
all  trial  maneuvers  for  comparison. 

In  general,  for  each  flight  the  four  sets  of  weights 
show  good  consistency,  but  little  or  no  consistency  is 
evident  across  the  four  flights  for  each  student.  As  a 
check  on  this  lack  of  consistency,  the  inequalities  for 
all  four  flights  for  each  student  pilot  with  reference 
flight  #14  were  combined  and  solved.  Student  pilots  1  and 
4  had  nonzero  weights  only  on  questions  1  and  9,  while  the 
other  student  pilots  had  nonzero  weights  on  questions  1 
and  2.  These  were  the  only  questions  on  which  the  respec¬ 
tive  pilots  had  essentially  unanimous  affirmative  answers 
over  all  flights. 


Basic  Difficulties  in  Method 

Since  they  will  be  referred  to  in  the  discussion  and 
analysis,  the  following  tables  are  introduced  here:  Table 
2,  Code  in  Octal  Cell  Number  for  Question  Number1;  Table  3, 


'At  each  decision  point,  the  student  pilot  and  each 
trial  maneuver  are  assigned  a  cell  number  which  is  a 
binary  number  in  the  computer  (but  printed  as  an  octal 
number)  with  a  1  in  each  binary  position  correspond i ng  to 
questions  with  a  yes  answer.  Therefore,  octal  number 
1342  would  indicate  yes  answers  to  questions  2,  6,  7,  8, 
and  10.  Codes  in  1342  are  2,  40,  100,  200,  and  1,000 
which  by  Table  2  correspond  to  questi ons  2,  6,  7,  8,  and 
10,  respectively. 
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TABLE  3 


Octal  Number  Pairs  Describing  Inequalities  with 
Reference  Run  #14  and  Best  Trial  Maneuvers 
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TABLE  3  (Concluded) 


File  13 


60 

1?0 

1  0 

?  o  ?  c 

0 

"  ?o 

1  0 

60 

0 

40 

110? 

44 

File 

M 

File 

16 

0 

1120 

0 

224 

6  0 

?1?U 

410 

3?0 

1  0 

l?i' 

4  6  0 

3320 

0 

42 

2010 

1324 

10 

6C 

100 

40 

0 

?0 

File  16 


File  17 


File  18 


»,  A 

1120 

5C 

2120 

1  0 

120 

0 

20 

0 

102 

4  C 

2112 

20 

2112 

4  0 

1102 

1020 

14 

100 

240 

60 

10 

4  r  o 

300 

2  0  00 

30  0 

0 

20 

:  o  .'0 

2200 

0 

3000 

<0 

3100 

0 

110 

20 

211  0 

c 

2000 

20 

4;  o 

4C0 

210 

File  19 


File  20 


40 

300 

1400 

300 

60 

11)0 

2??0 

1110 

0 

ICO 

0 

24< 

0 

10CO 

0  100 

0  u, 

loro  ?  o i r 

0  3000 
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Octal  Number  Pairs  Describing  Inequalities  with  Reference 
Run  #14  and  Best  Trial  Maneuver;  Table  4,  Position  of 
Student  Pilots  and  Instructor  Pilots  at  0,  10,  and  17 
seconds2;  Table  5,  Cell  Numbers  in  Octal  for  Decision 
Points  in  Student  Flights;  and  Ta bl e  6 ,  Genera  1  Description 
of  Student  Flights,  Comparisons  with  Reference  Flight  *14. 

While  the  score  for  a  student  pilot  at  a  decision 
point  is  determined  by  the  position  relative  to  the  refer¬ 
ence  yo-yo,  the  inequalities  which  are  used  in  determining 
the  weights  depend  on  both  the  position  of  the  student 
pilot  and  the  projected  position  of  the  trial  maneuvers  of 
the  AML.  For  example,  even  though  the  student  pilot  has  a 
negative  answer  to  a  question  at  all  decision  points,  if 
the  AML  does  likewise,  the  question  will  not  appear  in  the 
inequalities  and  will  receive  the  maximum  weight.  As  can 
be  seen  from  Table  5,  in  flight  #10,  questions  7  and  10 
(code  100  and  1,000)  never  had  an  affirmative  answer  at  a 
decision  point;  but  as  can  be  seen  from  Table  3,  the  codes 
did  not  appear  in  any  of  the  inequality  pairs  for  flight 
#10.  Hence,  none  of  the  best  trial  maneuvers  ever  had  an 
affirmative  answer  to  these  questions;  and  so  as  shown  in 
Table  1,  both  questions  had  a  weight  of  5. 

Unlike  the  AML,  the  student  pilot  can  choose  a  defin¬ 
itely  inferior  maneuver  whose  set  of  affirmative  answers 
is  a  subset  of  the  affirmative  answers  of  the  trial 
maneuvers.  This  results  in  the  first  number  of  the 
inequality  number  pair  being  zero,  so  zero  weights  have  to 
be  assigned  to  any  question  whose  code  appears  in  the 
second  number.  All  student  flights  had  at  least  one  such 
maneuver  at  some  decision  point  (reference  Table  3).  In 
over  half  the  flights,  the  zero  weights  had  a  cascading 
effect  in  that  codes  for  questions  with  zero  weights 
occurred  in  the  first  number  of  other  inequalities  making 
the  first  number  a  zero.  Extreme  cases  are  flights  #5  and 
#17.  In  both  cases,  all  questions  whose  codes  appeared  in 
the  second  number  of  an  inequality  pair  had  to  be  assigned 
weights  of  zero  (reference  Tables  1  and  3).  Flight  #17 
was  one  of  the  better  flights  (reference  Tables  5  and  7). 


•'These  are  actually  4,  14,  and  21  seconds  into  the 
flinht  as  the  program  at  Luke  at  the  time  of  student  pilot 
flights  did  not  start  recording  data  until  4  seconds  into 
the  flight.  The  times  on  the  instructor  pilot  data  were 
adjusted  to  match.  The  total  time  on  reference  flight  "14 
is  24  seconds  so  with  prediction  time  of  3  seconds  only  21 
seconds  can  be  used  by  the  AML  or,  after  adjustment,  17 
seconds  when  *14  is  used  as  reference. 


33 


35 


i 


mrooiNh.rsNrsNNrsronrorjrororo 

LniDLOLnLn^D^VD^iD^NLnLfiLni —  . —  r— 
«^-  <}•  cj"  c3"  lO  LO  LO  LO 


C\J  C\J  CVJ  CXI  i 


i —  i —  . —  co  ro  co  co 


mronnnNNrvNNNNNNrvN 
ininLDN'.O'DLD'DyD^DkD'DU'M^Lni — 
<^j-«3-«3-cj-,^t’^'*TLO«^«3-LOLOLO,=j'LOLO 

CVJCVI(OCOCOCVJC\JC\JC\JC\JCVJCO 


fONNNr->NNNKr\NNNNrsNNN 
inLf)UD^i£><X)y3VDU3LOrvU)VD^>vDl£>^)VD 
LO  OJ  OJ  OJ  OJ  OJ  OJ  C\J  OJ  OJ  C\J  C\J  C\J  OJ  OJ  OJ  lo 


OJ  CM  OJ  OJ  OJ  C\J 


fOOOr-f^nrONrsNf^NNNNlNNNN 

i^inin^^^DcvjNvDiD^Lninini/JLnLOLn 

#»r>  p  rv  n  »>  n  #s  »> 

cvj  -—  r—  rororof— r—  ro 


oocop-  cocorococooocooooocooooo< —  r—  .— 

LO  LO  LO  LO  LO  LO  LO  LT)  LD  LO  LO  LT)  LO  LO  LO  LO  LO  VO 


CVj  CVJ  OO  CSJ  CNJ  CNJ  CNJ  OJ  CNJ  CNJ 


rooorocorONNrNpNpxronrororonron 
lOLOLOLOLOLOLOLOLnuOLOLOLniOLOLOOJOJ 
^  UD  LO  <J-  LO  LO  LO 


OJOJOJOJOJOJCVJOJOJOJ 


ooroi —  roronooororororororororororoi — 

LO  LO  LO  LO  LO  LO  LO  LO  LO  LO  LO  LO  LO  i —  , —  , —  OJ  C\J 
^  ^  ^  LO  LO  LO  LO  LO  LO 


OJOJOJOJOJOJOJOJOsJOJOJ 


roroi —  cocooocococororocooorococococo 
LOLOLOLOLOLOlOlOLOlOLO. —  . —  i —  c\j  c\j  cvi  CVJ 
^^^^•^^^t^^^tLDimoLninLOirjLn 


CVJCVJCVJCVJCVICVJCVJCVJOdCVJCOCO 


■WCO^t  LOCDNCOC^O. —  CVJ  m  ^  LO  LO  r\ 


r 


vo 


CO 


(/)  r— 

■M  ^ 
-IZ 

CD  +-> 

•  i —  -C 
<—  CD 


c 

CD  CD 
-o  O 

Z3  C 
4->  CD 

c/>  l- 

t*.  * 

t:  w- 
°  a; 

c  i_ 

o 


Q-  *r- 

*<-  5 

U  ^ 
on  ~ 
CD  ^ 
Q  oo 

C 

I—  o 

rO  on 

i-  *i — 
CD 
C 
CD 
O 


S- 

03 

CL 

E 

o 

o 


CD 

U 

e 

03 

c 

■  r— 

s- 

•f- 

.0 

4-> 

0)  03 

CD 

CD 

O  CD 

03 

T3 

CD 

U  X3 

03  o 3 

CD 

CJ 

03  C 

s- 

CL 

CD 

03 

i-  13 

4->  CL 

3 

CL 

S- 

CD  D  O 

3 

OO 

4-> 

u  S-. 

“O 

XJ 

1 

03  XJ  CD 

CD  CD 

C  X> 

CD 

“O 

Sw  c 

CD  CJ  U 

3  CD 

+■> 

o 

c 

+->  zj  -a 

CJ  03  03 

O  4-> 

S- 

, — 

CD  CD 

3 

o  o 

h3  L  L 

S-  S- 

03 

CJ  CD 

o 

xj  s-  o 

4->  4J 

CD  03 

+-> 

03 

03  03 

s- 

C  CD  CD 

4-> 

00 

S-  s- 

CD 

3  i 

-a  xj 

</> 

>> 

4-> 

O  <D  l 

x>  c  c 

•r— 

P> 

_D 

s- 

to  >> 

CZ  3  3 

03  P-> 

3 

x>  X3 

o 

C7!  L  r- 

3  0  0 

<+-  3 

_o 

TD 

c  c 

o 

O  i- 

O  S-  S- 

1  X) 

CD 

3  3 

CL 

L  2  fU 

i-  CD  CD 

1 

CD 

O  O 

1 

O  1  CD 

CD 

>s  >> 

E 

O 

S-. 

1 

O  I 

s-  *u 

. —  p— 

•r~ 

, — 

CD  CD 

CD 

CL  CD  "O 

s-  o  o 

s- 

4-> 

i — 

4-> 

+->  C 

•p-  o  o 

03  03 

o 

L.  C. 

03 

>5^0 

03  CL  CD 

<D  03 

CD 

•p—  *r— 

i — 

^  > —  u 

4-  I  I 

E 

03  03 

CD  CD 

l  l  l 

00  00 

03 

>> 

<4-  «+- 

OO 

>  to  to 

t  CD  CD 

XJ  X) 

oo 

i — 

1  1 

~a 

1  "O 

CD  P  P 

c  c 

s- 

1  1 

c 

1  CZ  P~ 

4->  03  03 

o  o 

CD 

03 

o  o 

o 

o  o 

05  i —  i — 

o  U  <U  H  (U 

CD 

>> 

CJ 

>>  u  * 

r— 

CD  CD  CJ  +->  CJ 

1  1 

CD 

1  CD  X 

to  to 

on  co  03 

03 

00 

o  o 

to 

O  to  CD 

“C3  X  X» 

4->  S-  -O 

>>  >> 

CL 

C  C  C 

cm  ro  +-> 

03  +-> 

C 

CM 

CO  03 

O  o  o 

o 

T3  "CJ  CD 

CD  X3 

CJ  O  CJ 

X>  XJ  XJ  “O  X3 

u 

CD  CD  U 

•» 

U  CD  *•  Sw 

0)  CD  CD 

c  c  c: 

c  c 

CD 

CD  CD  03 

X 

03  CD  X  CD 

to  oo  to 

03  03  3 

03  =3 

to 

a  CL  L 

CD 

i-  CL  CD  x: 

O 

O 

CO  OO 

n. 

+->  oo  CL  CD 

r-  CO  CM 

S- 

S-  S- 

1  1 

03 

i  03 

CD  CD  CD  CD  CD 

cut! 

XJ  x:  x: 

#*  p>  p> 

-C  XT 

si 

CD  CD  SZ 

-C 

c:  cdx: 

XXX 

03  CD  i- 

CD  $*. 

X 

•r-  *r-  Z3 

CD 

3  *r—  CD  >•> 

CD  CD  CD 

•r—  *r*  *r— 

•r~  *r*“ 

CD 

re  o 

•r- 

O  XT  -f-  r— 

Q.  CL  CL 

JZ  -C  03 

x:  os 

CL 

x: 

x:  4J 

03  CO  CO 

»P- 

4- 

03 

>^  >>  CD 

cd  >>  x: 

X  X  I 

X  1 

X3  jQ 

>> 

-Q  >>  CD 

x:  _c  x: 

CD  CD  l 

CD  f 

"O 

i- 

i_  c  -p- 

C7>  CD  CD 

Q-  CL  +J 

CL  4-> 

O 

CD 

•0X3  0 

CD 

•P-  X3  CD  1— 

•i—  *r—  *r- 

03  03  C 

03  C 

r— 

U 

CD  CD  O 

> 

03  CD  >  oo 

rx:r 

CD 

CD 

03 

J  5  CJ! 

JZ  -C=  U  -C  U 

-£Z 

s- 

o  o 

JZ 

oxr 

x:  x:  x: 

+J  4->  V) 

+->  to 

-P> 

4-> 

r-  t—  SZ 

C  r-  P  -P 

p  P  p 

•f-  -r—  0) 

•r-  CD 

•r- 

. —  . —  4-J 

•p— 

r— 

•P“  *1—  ■!—  ' 

S  2  -O 

3:  -a 

?  *u 

O  O  -r- 

•»-  o  ?  ^ 

?  3  3 

C 

c+-  5 

5  *4— 

o  o  <♦- 

O  M— 

o 

3 

o 

o  o 

o  o  o 

>,  >,  O 

>>  o 

>)  O 

o  o  o 

>> 

o  o  >^  >> 

>^  >^  >> 

i  I 

1 

1 

S_ 

>>>>>> 

1 

>)  >^  1  1 

1  1  1 

o  o  <u 

O  (D 

o 

CD 

1  1  1 

o 

I  l  o  o 

o  o  o 

>>>— 

>, 

O  O  O 

o  o  >,  >, 

>> 

~7 

x> 

S- 

>>>>>> 

>>  >> 

■OT3'DT3‘OT3 

•i— 

X3 

XJ  “D 

“0X3X3 

CD  <D  t- 

<D  *»- 

CD 

03 

“0X3X3 

CD 

X3  XJ  CD  CD 

CD  CD  CD 

CD  CD  E 

CD  E 

CD  v*- 

CD  CD  CD 

CD 

CD  CD  CD  CD 

CD  CD  CD 

CL  Q- 

CL 

CL 

1 

CD  CD  CD 

CL 

CD  <D  CL  CL 

CL  CL  CL 

OO  CO  CD 

oo  CD 

to 

1 

CL  CL  CL 

to 

CL  CL  tO  OO 

00  fO  to 

1  1  -SZ 

\  -C 

1 

O 

i/>  to  OO 

1 

to  to  l 

t  1  1 

n  x:  -m  x  -p  x: 

>> 

1  1  1 

-C 

i  i  x:  x: 

jC  -C  ^ 

CD  CD 

CD 

CD 

1 

?  5  5 

CD 

5  5  CD  CD 

CD  CD  03 

•r-  -f-  C 

C 

•r- 

o 

o  o  o 

•r— 

O  O  •«-  -r- 

•r—  *p“  *r— 

X  X  *■“ 

X  -r- 

X 

>> 

_ 1  _ f _ 1 

zz 

- 1  - 1  I  I 

f—  CM 

co 

ID  CD  CO 

CD  O  i —  CM 

ro  tr> 

N  CO  O 
r—  r—  r— *  C\J 


37 


High-speed  yo-yo  with  high  apex,  5  seconds  late--poor  ground  trace 

High-speed  yo-yo  with  same  apex,  3  seconds  early--fair  ground  trace 

High-speed  yo-yo  with  same  apex  at  same  time--fair  ground  trace 

High-speed  yo-yo  with  same  apex,  2  seconds  early--good  ground  trace 

High-speed  yo-yo  with  high  apex,  1  second  early--fair  ground  trace. 


TABLE  7 

Number  of  Affirmative  Answers  per  Question 
with  Reference  Flight  #14 

Maximum  number  per  question  is  18  except  for  file  19, 
which  has  a  maximum  of  16  per  question. 


File 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

1  1 

1 

18 

17 

0 

9 

9 

5 

14 

0 

18 

3 

5 

2 

18 

16 

0 

10 

8 

13 

6 

0 

18 

1 

1 

3 

18 

14 

0 

13 

7 

7 

1  2 

0 

18 

4 

6 

4 

18 

17 

0 

9 

9 

13 

6 

0 

18 

0 

6 

5 

18 

18 

1  6 

1 1 

7 

14 

5 

5 

14 

3 

4 

6 

18 

18 

17 

11 

8 

14 

5 

10 

11 

5 

1  2 

7 

18 

18 

16 

9 

9 

14 

6 

6 

1  5 

3 

11 

8 

18 

18 

12 

15 

3 

18 

0 

0 

18 

2 

9 

9 

18 

18 

15 

8 

10 

1  5 

4 

14 

4 

2 

9 

10 

18 

18 

1  6 

10 

8 

18 

0 

10 

8 

0 

4 

11 

18 

18 

17 

1  5 

5 

18 

0 

16 

2 

0 

3 

12 

18 

17 

0 

12 

6 

6 

13 

4 

18 

5 

7 

13 

18 

1  7 

0 

14 

4 

11 

8 

0 

18 

2 

12 

14 

18 

16 

0 

1  6 

2 

13 

6 

0 

18 

0 

11 

15 

18 

18 

5 

14 

4 

1  6 

5 

0 

18 

3 

10 

1  6 

18 

14 

0 

1  6 

2 

18 

0 

0 

18 

0 

9 

17 

18 

17 

12 

10 

8 

16 

3 

0 

18 

8 

4 

18 

18 

18 

17 

4 

1  5 

18 

0 

1  7 

3 

0 

6 

1  9 

16 

1  6 

1 1 

8 

9 

1  5 

4 

3 

1  6 

4 

1  2 

20 

18 

18 

8 

1  2 

7 

1  5 

4 

0 

18 

10 

7 
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The  following  analyses  were  made: 

1.  Comparison  of  overall  flightpaths  with  weights 
for  possible  correlations. 

2.  Applicability  of  weights  as  measures  of  the 
importance  attributed  by  the  pilot  to  the 
respective  questions. 

3.  Possibility  of  the  AML  using  derived  weights 
serving  as  a  model  of  the  student  pilot. 


Comparisons  of  Flights  with  Weights 

As  can  be  seen  from  Tables  4  and  6,  not  all  flights 
were  high-speed  yo-yo  maneuvers  and  those  that  were  showed 
considerably  variability.  The  weights  of  flights  of  the 
same  type  were  studied  for  possible  use  as  identifiers  of 
the  flight  types.  No  reasonable  identifiers  were  found. 
For  example,  flights  #1,  #12,  and  #20  were  high-speed  yo¬ 
yo  maneuvers  but  have  quite  different  weights.  Similar 
results  hold  for  flights  #7  and  #9  which  were  low-speed 
yo-yo  maneuvers  and  for  flights  #5,  #6,  and  #10  each  of 
which  was  a  low-speed  yo-yo  followed  by  a  high-speed  yo-yo. 
From  a  different  view  point,  flights  #3  and  #4  have  same 
weights  in  relation  to  reference  flight  #22  but  are  quite 
different  flights  types.  A  similar  result  holds  for 
flights  #10  and  #12  with  reference  flight  #14  and  best 
trial  maneuver. 


Weights  as  Measures  of  Importance 

If  the  weight  is  to  measure  the  importance  attributed 
by  the  pilot  to  a  question,  the  number  of  affirmative 
answers  to  each  question  must  be  involved  in  the  judgment 
criterion.  Now,  for  all  practical  purposes  the  assignment 
of  weights  is  dichotomous  in  that  either  the  maximum  value 
of  5  or  the  minimum  value  of  0  is  assigned  in  almost  all 
cases.  This  seems  to  indicate  a  threshold  as  criterion; 
i.e,  if  the  percentage  of  affirmative  answers  is  above  the 
threshold,  a  weight  of  5  should  be  assigned,  otherwise,  a 
weight  of  0.  From  Table  5,  Table  7  was  constructed 
showing  the  number  of  affirmative  answers  per  question  for 
each  flight  with  reference  flight  #14.  Using  this  table 
in  conjunction  with  Table  1,  the  percentages  of  correct 
nonzero  weights,  of  correct  zero  weights,  and  of  correct 


total  weights  were  computed  for  two  thresholds,  one  of  67 
per  cent  and  the  other  of  50  per  cent.  The  results  are 
shown  in  Table  3.  These  results  show  a  large  variance 
over  the  flights  with  an  average  total  correct  for  each 
threshold  of  around  66  per  cent.  As  would  be  expected, 
the  lower  threshold  increased  the  percentage  correct  of 
nonzero  weights  but  decreased  the  percentage  for  zero 
weights  correct.  A  random  selection  would  give  around  50 
per  cent  correct  so  the  67  per  cent  correct  is  not  appre¬ 
ciably  better  than  random.  Moreover,  the  per  cent  correct 
varies  over  the  different  flights  from  46  to  82.  This 
would  appear  to  rule  out  the  assigned  weights  as  good 
measures  of  the  importance  attributed  by  the  pilots  to  the 
different  questions. 


AML  as  Model  of  Student  Pilot 

In  Phase  I  of  this  study,  the  minimum  and  maximum 
solution  gave  limits  on  possible  solutions  for  the  set  of 
weights.  Any  such  solution,  if  used  as  weights  in  the  AML 
program,  would  cause  it  to  fly  the  observed  engagement 
exactly  as  the  observed  AML  program  did.  If  the  weights 
obtained  for  a  student  pilot  against  a  reference  flight 
were  used  in  the  AML  flying  against  the  same  reference 
flight,  would  the  AML  fly  the  same  pattern  as  the  student 
did?  Most  likely,  it  would  not.  As  previously  reported, 
different  student  flights  had  the  same  set  of  weights,  but 
their  flight  patterns  were  very  different.  Obviously,  the 
AML  cannot  model  both.  Also,  when  the  AML  was  given  zero 
weights  for  two  questions,  it  flew  a  high-speed  yo-yo 
against  one  reference  flight  but  a  low-speed  yo-yo  against 
the  other.  Further,  at  many  decision  points,  the  cell 
number  resulting  from  student  pilots'  maneuvers  was  not  a 
cell  number  resulting  from  one  of  the  trial  maneuvers. 
Hence,  there  is  no  guarantee  that  the  best  trial  maneuver 
for  the  AML  would  adequately  approximate  the  student 
pilot's  maneuver  at  each  decision  point.  Actually,  at  any 
decision  point,  it  is  unlikely  that  the  student's  maneuver 
would  be  exactly  one  of  the  trial  maneuvers  increasing  the 
probability  of  large  accumulated  errors  in  the  flight's 
parameters . 


Other  Anomalies 

As  can  be  seen  from  Tables  4  and  6,  student  flights 
#10  and  #11  were  very  poor  flights,  but  both  have  most 
weights  nonzero,  particularly  flight  #10.  In  student 
flights  #10  and  #11,  the  values  of  the  parameters  are  far 
from  the  boundaries  of  the  envelope  of  parameter  values 
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around  the  reference  parameter  values.  As  a  result,  the 
cell  numbers  of  the  student  pilot  maneuver  as  well  as 
those  of  the  trial  maneuvers  varied  very  little  during  the 
flight.  Hence,  few  inequalities  were  generated  or,  equiv¬ 
alently,  few  questions  had  restrictions  on  their  weights. 
Thus,  most  of  the  questions  received  weights  of  5. 

Surprisingly,  a  very  good  high-speed  yo-yo  would 
probably  not  give  rise  to  many  inequalities;  for  example, 
a  high-speed  yo-yo  whose  parameter  values  were  always 
within  the  envelope  of  parameter  values  of  the  reference 
flight  would  have  yes  answers  to  all  questions.  Hence, 
there  would  be  no  questions  peculiar  to  the  trial  maneu¬ 
vers,  and  the  second  number  in  the  inequality  pair  would 
always  be  zero.  The  resulting  inequality  is  simply  that 
the  sum  of  the  weights  for  questions  peculiar  to  the  good 
yo-yo  is  greater  than  or  equal  to  zero.  But  this  is 
already  part  of  the  LP  program  restrictions,  and  the  pair 
would  be  disregarded.  However,  a  good  yo-yo  would  have 
consistently  high  scores  (10  or  11  out  of  11)  while  a  poor 
one  would  not. 

As  previously  shown  in  the  report,  the  AML  with 
weights  all  equal  to  5  flies  a  high-speed  yo-yo  which  is  a 
good  approximation  to  the  reference  yo-yo.  Hence,  it 
would  model  the  good  high-speed  yo-yo. 


Application  of  a  Modified  AML 

The  data  in  Table  5  are  descriptive  of  aspects  of  the 
student  pilot's  performance,  some  relative  to  the  target 
aircraft,  and  others  relative  to  the  reference  flight. 

They  are  a  history  of  his  performance  relative  to  the 
question  set;  for  example,  the  results  for  flights  #1 
through  #4  show  that  student  pilot  #1  consistently  stayed 
behind  the  target  (question  1,  code  =  1 ) ,  almost  always 
had  the  target  in  sight  (question  2,  code  =  2 ) ,  but  was 
always  in  view  of  the  target  (question  3,  code  =  4 ) . 

The  derivation  of  the  data  in  this  table  does  not 
involve  the  part  of  the  AML  program  which  determines  the 
next  AML  maneuver  at  each  decision  point  and  flies  the  AML. 
Hence,  the  usual  restriction  that  the  set  of  questions  be 
amenable  to  that  part  of  the  AML  logic  which  determines 
the  next  maneuver  no  longer  holds.  A  set  of  questions 
can  be  used  which  involves  those  aspects  of  the  flight 
which  are  of  interest  to  the  instructor. 
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The  modified  version  (actually  a  stripped  down 
version)  of  the  AML  can  determine  and  store  these  data 
for  later  extraction  and  display. 


I  V .  CONCLUSION 


In  Phase  I,  the  basic  methodology  was  developed  for 
computing  the  weights  from  recorded  flight  data.  Here  in 
Phase  II,  the  approach  was  refined  based  on  the  experience 
in  Phase  I  and  was  evaluated  using  real  world  data 
recorded  on  the  SAAC  at  Luke  AFB.  The  evaluation  revealed 
the  following  critical  problems  in  this  approach. 


AML  and  LP 

The  refined  AML  program  was  run  against  the  student 
flight  data  to  determine  in  conjunction  with  a  linear 
programming  (LP)  program  the  appropriate  weights  of  the 
student  pilot  with  respect  to  the  question  set.  In  all 
cases,  the  resulting  inequalities  required  that  some  of 
the  weights  be  zero  in  order  to  obtain  a  solution;  thus, 
the  restriction  that  the  weights  be  between  1  and  5  had 
to  be  dropped.  Weights  were  still  restricted  to  be  less 
than  or  equal  to  5,  and  the  usualy  requirement  of  nonnega¬ 
tive  weights  remained.  The  result  was  that  the  sets  of 
weights,  with  two  exceptions,  took  on  only  the  values  0 
and  5.  This  eliminated  any  tight  bounds  on  solutions 
except  for  the  zero  weights. 


Basic  Problems  with  Method 

The  inequalities  derived  to  determine  the  set  of 
weights  depend  on  the  relative  position  of  the  student 
flight  and  the  AML  trial  maneuvers  at  each  decision  point. 
Hence,  they  do  not  always  reflect  the  position  of  the 
student  relative  to  the  reference  yo-yo.  Also,  the  possi¬ 
bility  plus  the  actual  occurrences  of  definitely  inferior 
maneuvers  by  the  student  pilots  resulted  in  zero  weights 
having  to  be  allowed.  Hence,  when  an  inferior  maneuver 
occurred,  the  zero  weights  were  determined  by  a  single 
decision  point.  In  addition,  there  was  often  a  cascading 
effect  of  zero  weights  when  other  inequalities  had  the 
code  for  questions  with  zero  weight  in  the  first  word  of 
the  inequalities. 


Weights  V erjs  u s  Flight  Patterns 

The  possibility  was  studied  of  correlating  with  sets 
of  weights  the  different  types  of  flight  patterns  as 


described  in  Table  7  .  No  such  correlations  were  found. 
Similar  flights  had  different  weight  sets,  and  cases  of 
completely  different  flights  with  the  same  weight  sets 
were  found. 


Weights  Versus  Frequency  of  Affi  rma  ti  ve  Answers  to  Ques  t  i  oris 

To  be  meaningful,  the  weights  should  reflect  the 
importance  attributed  to  each  corresponding  question.  On 
the  other  hand,  the  frequency  with  which  each  flight 
performance  question  has  an  affirmative  answer  should  be 
a  measure  of  its  importance  and,  thus,  should  correlate 
with  the  corresponding  weight.  For  all  practical  purposes, 
the  weights  are  dichotomous ,  being  either  0  and  5,  thus 
suggesting  a  threshold  logic.  The  percentage  agreement 
between  the  weights  and  frequencies,  for  frequency 
thresholds  of  50  per  cent  and  67  per  cent,  was  computed 
for  the  20  student  flights.  The  percentage  agreement 
varied  considerably  over  the  different  flights  for  both 
thresholds  with  a  66  per  cent  agreement  overall  in  both 
cases.  The  large  variability  over  the  different  flights 
plus  the  low  percentage  of  agreement  ruled  out  use  of  the 
weights  as  measures  of  the  importance  attributed  by  the 
pilot  to  the  various  questions. 


Application  of  Modified  AML 

Now,  for  each  student  flight  the  sequence  of  octal 
cell  numbers  representing  the  answers  to  the  question  set 
at  each  decision  point  is  a  history  of  his  performance 
relative  to  the  criteria  {the  question  set).  This  history 
is  in  a  form  amenable  to  automatic  extraction  of  data  and 
analysis.  Hence,  since  the  requirement  that  the  set  of 
questions  be  amenable  to  flying  the  AML  would  not  be  a 
useful  implementation,  questions  could  be  developed  which 
involve  those  aspects  of  the  flight  of  interest  to  the 
instructor.  The  modified  AML  program  can  then  score  the 
student  flight  at  each  decision  point  and  store  the 
sequence  of  scores  for  later  data  extraction  and  analysis. 


Summary 

In  summary,  a  method  that  looked  relatively  promising 
during  initial  tests  using  simulated  data  failed  when  real 
world  data  from  the  SAAC  were  used.  There  is  no  evidence 
that  the  method  is  useful  here,  and  some  serious  problems 
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were  discovered  concerning  the  methods  employed  in  the 
approach.  Based  on  this  analysis,  no  further  pursuit  into 
this  effort  is  recommended. 
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V.  APPENDIX  A 


Evaluation  of  Original  AML  in  Flying  High-Speed  Yo-Yo 
Maneuvers 

Since  in  the  past  the  original  Adaptive  Maneuvering 
Logic  (AML)  was  not  observed  to  fly  a  high-speed  yo-yo,  it 
had  been  assumed  that  it  would  not.  To  verify  this,  the 
original  AML  was  flown  against  the  target.  In  the  first 
run,  the  AML  kept  essentially  the  same  altitude  but  swung 
well  outside  the  path  of  the  attacker  and  then  came  back 
in  behind,  sort  of  a  horizontal  yo-yo;  however,  it  was 
pointed  out  that  this  is  not  a  good  maneuver  since  the 
target  could  become  the  attacker  and  take  away  the  posi¬ 
tional  advantage  of  the  AML.  The  ground  trace  is  shown  in 
Figure  A-l.  It  was  noticed  that  the  range  and  range  rates 
for  the  first  decision  point  were  about  10  times  as  high 
as  they  should  be.  An  error  was  found  in  that  part  of  the 
program  which  saves  the  last  three  positions  of  the  target 
in  order  to  project  its  position  at  decision  time.  At  the 
first  decision  point,  only  two  points  had  been  saved. 

This  was  corrected,  and  the  original  AML  flown  against  the 
target.  As  shown  in  Figures  A-2  and  A-3,  a  completely 
different  path  was  flown  but  still  not  a  high-speed  yo-yo. 
It  used  throttle  settings  to  adjust  velocity.  It  is 
suspected  that  it  also  used  speed  brakes,  but  that  is  not 
a  printout  item  and  cannot  be  verified  without  additional 
program  modification.  The  result,  of  course,  ruled  out 
using  the  original  AML. 


Performance  of  Revised  AML  on  High-Speed  Yo-Yo 

A  subroutine  In  the  AML  program  evaluates  each  candi¬ 
date  trial  maneuver  of  the  attacker  using  the  question 
set.  This  subroutine  was  rewritten  using  the  new  set  of 
questions.  The  new  AML  (i.e.,  using  the  new  set  of  ques¬ 
tions)  did  fly  a  high-speed  yo-yo,  but  It  did  not  produce 
as  high  an  apex  as  the  reference  runs.  It  closed  behind 
the  target  sooner  and  at  a  higher  closing  rate.  Figures 
A-4  and  A-5  represent  a  flight  using  the  new  AML  with  run 
022  as  reference. 

Five  runs  were  made  using  Luke  flight  014  as  the 
reference  flight,  and  five  were  made  using  Luke  flight  #22 
as  the  reference.  In  the  runs,  different  sets  of  weights 
and/or  different  tilt  angles  were  used.  (In  the  trial 
maneuvers,  only  multiples  of  the  tilt  angle  are  allowed  as 
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angles  for  the  maneuver  planes.)  Tilt  angles  of  10 
degrees  or  15  degrees  were  used.  The  weights  and  tilt 
angles  for  each  of  the  five  runs  are  given  in  Table  A  - 1 . 
Both  sets  of  five  runs  used  the  same  sets  of  weights  and 
tilt  angles. 

Each  of  the  five  runs  with  the  same  reference  run 
followed  the  general  pattern  of  the  reference  run  and  did 
not  display  any  distinctive  differences  among  themselves. 
Each  flight  did  a  high-speed  yo-yo  with  the  apex  occurring 
at  the  same  time  as  the  reference  yo-yo  but  at  a  lower 
altitude.  Also,  each  flight  after  the  apex  homed  in  more 
quickly  on  the  target  and  reached  the  target's  altitude 
sooner  than  did  the  reference  yo-yo. 

Next,  a  more  drastic  change  was  made  in  the  weights. 
The  weights  for  the  first  two  parameters,  altitude  and 
altitude  rate,  were  set  to  zero;  flights  were  made  with 
both  runs  #14  and  #22  as  reference.  The  results  were 
quite  different.  The  flight  with  reference  run  #14  was  a 
high-speed  yo-yo  with  slightly  lower  apex  than  previous 
flights  but  better  ground  trace.  Plots  are  given  in 
Figures  A-6  and  A-7.  The  flight  with  reference  run  #22 
was  a  low-speed  yo-yo  with  a  poorer  ground  trace.  Plots 
are  given  in  Figures  A-8  and  A-9. 

A  more  careful  study  of  the  output  data  showed  that 
in  all  trial  maneuvers,  the  velocities  were  the  same  at  a 
decision  point  so  that  this  parameter  di d  no d i fferent i a ti ng . 
An  analysis  of  the  program  revealed  that  velocity  is  not 
updated  during  the  trial  maneuver,  so  the  end  velocity  for 
each  trial  maneuver  is  the  same  as  at  the  start.  Hence, 
it  is  the  same  for  all  trial  maneuvers;  however,  the  rate 
of  change  of  the  specific  energy  is  updated  as  is  the 
altitude.  The  specific  energy  at  the  end  of  the  trial 
maneuver  can  then  be  calculated  as  the  specific  energy  at 
the  start  plus  the  rate  of  change  times  the  prediction 
time.  Then,  since  specific  energy  is  equal  to  the  altitude 
plus  the  quotient  of  the  velocity  divided  by  2g,  the  velo¬ 
city  at  the  end  of  the  trial  maneuver  can  be  completed. 

This  change  was  incorporated  into  the  program. 
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Figure  A-8  (Concluded) 
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Figure  A-9.  Altitude  plot.  AML  versus  LUKE  #22. 

Weights  0,  0,  3,  5,  3,  1,  5,  1. 

Tilt  angle  15°. 
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